Technical Presentation 3: Verification and Fit

Ernesto Carrella

January 18, 2016

Review Panel Discussion

  • What are the sources of uncertainty in the model that are most concerning to you?
  • Does the methodology provide acceptable estimates of uncertainty? What additional sensitivity tests would you recommend?
  • Do you have recommendations on data sources that might be helpful to validate/inform input parameters or agent logic?
  • Are there any conceptual patterns that you think are missing?
  • For the West Coast:
    • What time series need to be hindcasted to be credible?
    • What patterns need to be reproduced for the West Coast to be credible?

Two main topics

  • Can we trust the model ?

  • Can we parametrize the model ?

Model Validation

Cautionary Tale: World3

Active Nonlinear Tests

  • Explore the parameter space for combinations that produce the opposite result
  • Modify the parameters only slightly
  • Optimization with “inverted” score

World3 ANTs

Pattern Oriented Modelling

What observed patterns seem to characterize the system and its dynamics, and what variables and processes must be in the model so that these patterns could, in principle, emerge?

When designed to reproduce multiple patterns, models are more likely to be “structurally realistic”

Active Nonlinear Tests on Patterns

  • They all pass at 10% and 20% variations
  • We will allow changes of up to 1000%

Front formation - pattern

Front formation - test

Parameter Default Flat
gas price 0.01 10

Front formation - demo

Hyperstability - pattern

Hyperstability - test

Parameter Hyper-stability Hyper-depletion Negative Relation
\(\epsilon\) 0.2 0.30 0.2
\(K\) 5000 8689.205965 7500
\(m\) 0.001 0.095023 0.085
hold size 100 830 800
cell width 10 7.25 3.75
gas price 0.01 3.194378 0.38
speed 5.0 1.590745 0.01
catchability 0.01 0.15 0.15

Hyperstability - demo

Hyper-depletion - demo

Oil Price - pattern

A plot of average distance to port per simulation after fixing oil prices and technology. The stepwise shape of the `movement only` fisher depends on the interaction of fish distribution, oil prices and cell size

A plot of average distance to port per simulation after fixing oil prices and technology. The stepwise shape of the movement only fisher depends on the interaction of fish distribution, oil prices and cell size

Oil Price - test

Parameter Original No Adaptation
\(\epsilon\) 0.2 0.27
\(K\) 5000 17004
hold size 500 978
cell width 1 13.27
speed 5.0 3.64
catchability 0.01 0.033
# of fishers 100 143

Fishing the Line - pattern

Fishing the Line - test

Parameter Default No fishing the line
\(m\) 0.001 0

Fishing the Line - demo

Switching Gear - pattern

A sample run where agents are allowed to switch gear and by consequence target species. Fishers tend to respond to variation in relative biomass distribution even without knowing it by following the example of those that are more profitable

A sample run where agents are allowed to switch gear and by consequence target species. Fishers tend to respond to variation in relative biomass distribution even without knowing it by following the example of those that are more profitable

Switching Gear - test

Parameter Small Difference Large Difference
gas price 0.01 10
\(\epsilon\) 0.2 0.8
\(K\) 5000 20000
\(m\) 0.001 0.2
hold size 100 160
cell width 10 20
speed 5.0 14.68
catchability 0.01 0.2
\(\epsilon_{\text{gear}}\) .2 .05

Switching Gear - demo

The dynamics generated by the active non-linear test.

The dynamics generated by the active non-linear test.

Directed Technological Change - pattern

Each line represents the average fuel inefficiency for an indpendent simulation. When facing free gas there is no incentive to improve fuel efficiency and therefore technology on average follows a random walk. The more expensive gas gets the more pronounced the march towards better gear becomes

Each line represents the average fuel inefficiency for an indpendent simulation. When facing free gas there is no incentive to improve fuel efficiency and therefore technology on average follows a random walk. The more expensive gas gets the more pronounced the march towards better gear becomes

Directed Technological Change - test

Parameter Adaptation Random Walk
gas price 1 1
\(\epsilon\) 0.2 0.48
\(K\) 5000 7242.33
\(m\) 0.001 0.0395
hold size 100 600
cell width 10 20
speed 5.0 0.1
catchability 0.01 0.2
\(\epsilon_{\text{gear}}\) .2 .70
\(\delta_{\text{gear}}\) .05 .48
max days at sea 5 Unlimited

Directed Technological Change - demo

Each line represents the average fuel inefficiency for an indpendent simulation. Even though the agents are facing high gas prices the biology and the map are structured in such a way as to make movement towards better gear unattractive

Each line represents the average fuel inefficiency for an indpendent simulation. Even though the agents are facing high gas prices the biology and the map are structured in such a way as to make movement towards better gear unattractive

One Species - pattern

One Species - test

  • 20% test: reduce correlation form -.9 to -.2
    • Lower gas price
    • low speed, large cell size
    • high catchability, low hold size
  • Parameter Default No correlation
    gas price 0.01 0

Quota Location - pattern

The normalized number of tows for each map cell over 5 simulated years for both the scenario with ITQ and TAC policy in place. The dashed line represents the divide between blue and red species at y=24. Any cell on the dashed line and below contains only blue fish (the bycatch) while the cells strictly above the dashed line contains only red fish

The normalized number of tows for each map cell over 5 simulated years for both the scenario with ITQ and TAC policy in place. The dashed line represents the divide between blue and red species at y=24. Any cell on the dashed line and below contains only blue fish (the bycatch) while the cells strictly above the dashed line contains only red fish

Quota Location - test

Parameter Original No targeting
\(\epsilon\) 0.2 0.05
\(K\) 5000 17314
hold size 100 10
cell width 10 1
speed 5.0 20
catchability 0.01 0.001
gas price 0.01 10

Quota Location - demo

Quota Gear - pattern

Quota Gear - test

Parameter Switch to red gear Switch to blue gear
\(\epsilon\) 0.2 0.05
\(K\) 5000 20000
\(m\) 0.001 0.07
hold size 100 10
cell width 10 20
speed 5.0 15
gas price 0.01 0.85

Quota Gear - demo

Fit to data

Osmose WFS

Osmose WFS

Osmose WFS

  • Ecosystem model
  • Calibrated on fixed mortality
  • We want to model the grouper fishers
  • We have logbook data and logit fits

Target Heatmap

Heatmap EEI

Area differences

Error minimization

Weaknesses of histogram matching

  • Weak to swaps
  • Not informative

Indirect inference

first step

Fitting problem

  • You have ABM with parameters \(\theta\)
  • You have real observations \(x_1,x_2,\dots,x_n\)
  • You’d like to fit by maximum likelihood \[\hat \theta = \max p(x_1,\dots,x_n|\theta)\]
  • Hopeless

Indirect inference

  • Use auxiliary model \(\beta(\cdot)\) to fit the data \[ \hat \beta = \beta (x_1,\dots,x_n) \]
  • Use ABM to generate synthetic data \[ x_1(\theta), \dots x_n(\theta) \]
  • Fit auxiliary model to synthetic data \[ \tilde \beta (\theta) = \beta ( x_1(\theta), \dots x_n(\theta)) \]
  • Minimize distance between the parameters of the auxiliary models \[ \theta = \min \left( \hat \beta - \tilde \beta(\theta) \right)^2 \]
  • Use logit regressions as our auxiliary model

Robustness of indirect inference

  • Minor assumptions on auxiliary model
    • Can be mispecified
    • Doesn’t have to predict very well
    • Needs to converge on \(\theta\)
    • Needs to be invertible
  • If \(\beta (\theta)\) differentiable, errors are gaussian
  • Distance over parameters is informative

Indirect inference and pattern oriented models

  • Assume logit regression as a “description” of a pattern
  • Indirect inference is a statistically explicit way of describing how far from a pattern we are

What I need from you

  • Is there any conceptual pattern that you think is missing?
  • West Coast Checklist:
    • What time series need to be hindcasted to be credible?
    • What patterns need to be reproduced for west coast to be credible?

Review Panel Discussion

  • What are the sources of uncertainty in the model that are most concerning to you?
  • Does the methodology provide acceptable estimates of uncertainty? What additional sensitivity tests would you recommend?
  • Do you have recommendations on data sources that might be helpful to validate/inform input parameters or agent logic?
  • Are there any conceptual patterns that you think are missing?
  • For the West Coast:
    • What time series need to be hindcasted to be credible?
    • What patterns need to be reproduced for the West Coast to be credible?